Systems and methods to overlay remote and local video feeds

ABSTRACT

According to some embodiments, a studio video feed, including a studio subject, may be received from a studio video camera. A remote video feed, including a remote subject, may be received from a remote video camera. The remote video feed may include, for example, the remote subject positioned in from of a solid-colored background. The remote subject may be overlaid into the studio video feed to produce a composite video signal, and at least one of the studio video feed and the remote video feed may be automatically adjusted to create an impression that the studio subject and the remote subject occupy a shared physical space.

FIELD

The present invention relates to systems and methods to combine remoteand local video signals. Some embodiments relate to systems and methodsfor efficiently overlaying remote and local video feeds.

BACKGROUND

A broadcast program might simultaneously include video information fromtwo different physical locations. For example, the program might includevideos of both (1) an interviewer (e.g., a program host located in abroadcast studio) and (2) a subject who is being interviewed (e.g.,located at sports stadium remote from the broadcast studio). Typically,each video is displayed in a separate box on the broadcast display. Forexample, a first box might display the face of the interviewer (e.g.,and the first box might be labeled “ESPN® Studios”) while a second boxmight display the face of the subject who is being interviewed (and thesecond box might be labeled “Fenway Park”). In some cases, a“split-screen” display might be provided (e.g., with the left halfdisplaying a studio video feed and the right half displaying a remotevideo feed). Such approaches, however, re-enforce the impression thatthe interviewer and subject are not occupying a shared physical spacewhich can distract and/or disorient viewers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a display including information from alocal video camera and a remote video camera.

FIG. 2 is a flow chart of a method in accordance with some embodimentsof the present invention.

FIG. 3 is a block diagram of a system in accordance with someembodiments of the present invention.

FIG. 4 is an illustration of a display including information from alocal video camera and a remote video camera in accordance with someembodiments.

FIG. 5 is a block diagram of a rendering engine in accordance with someembodiments of the present invention.

FIG. 6 is a tabular representation of a portion of a lookup table inaccordance with some embodiments of the present invention.

FIG. 7 illustrates a system in accordance with some embodiments of thepresent invention.

DETAILED DESCRIPTION

Applicants have recognized that there is a need for methods, systems,apparatus, means and computer program products to efficiently overlayremote and local video feeds. Note that a broadcast program mightsimultaneously include video information from two different physicallocations. For example, FIG. 1 is an illustration 100 of a display 130including a local video feed received from a local or “studio” videocamera 110, the local video feed including a local subject 112 (e.g.,the host of a program). The display 130 also includes a remote videofeed, from a remote video camera 120, the remote video feed including aremote subject 122 (e.g., a person being interviewed by the host). Inthis example, an image 142 of the local subject 112 is displayed in afirst box 140 labeled “Studio” while an image 152 of the remote subject122 is displayed in a second box 150 labeled “Remote” on the display130. Such an approach, however, re-enforces the impression that thelocal subject 112 and remote subject 122 are not occupying a sharedphysical space which can distract and/or disorient viewers.

To help avoid such a result, FIG. 2 illustrates a method that might beperformed, for example, by some or all of the elements described herein.The flow charts described herein do not imply a fixed order to thesteps, and embodiments of the present invention may be practiced in anyorder that is practicable. Note that any of the methods described hereinmay be performed by hardware, software, or any combination of theseapproaches. For example, a computer-readable storage medium may storethereon instructions that when executed by a machine result inperformance according to any of the embodiments described herein.

At 202, a local video feed is received from a local video camera, thelocal video feed including a local subject. As used herein, the phrase“video feed” may refer to any signal conveying information about amoving image, such as a High Definition-Serial Data Interface (“HD-SDI”)signal transmitted in accordance with the Society of Motion Picture andTelevision Engineers 292M standard. Although HD signals may be describedin some examples presented herein, note that embodiments may beassociated with any other type of video feed, including a standardbroadcast feed and/or a 3D image feed.

At 204, a remote video feed is received from a remote video camera, theremote video feed including a remote subject. The remote video feedmight comprise, for example, an HD-SDI signal received through a fibercable and/or a satellite transmission. According to some embodiments,the remote subject is situated in front of a solid-colored background(e.g., a “greenscreen”).

Note that the local and remote video cameras may be any device capableof generating a video feed, such as a Vinten® studio (or outside)broadcast camera with a pan and tilt head. According to someembodiments, at least one of the local video camera and the remote videocamera are an “instrumented” video camera adapted to providesubstantially real-time information about dynamic adjustments being madeto the instrumented video camera. As used herein, the phrase “dynamicadjustments” might refer to, for example, a panning motion, a tiltingmotion, a focal change, and/or a zooming adjustment being made to avideo camera (e.g., zooming the camera in or out).

At 206, the remote video feed and the local video feed are overlaid toproduce a composite video signal, wherein at least one of the localvideo feed and the remote video feed are automatically adjusted tocreate an impression that the local subject and the remote subjectoccupy a shared physical space. For example, the remote video feed mightbe automatically adjusted based on dynamic adjustments being made to thelocal video feed (e.g., the local camera might be slowly panning acrossa studio set). As a result of the automatic adjustment, the overlaidvideo feeds may create the impression that the remote subject is sittingnext to the local subject in a broadcast studio. As another example, theoverlaid video feeds might instead create the impression that the localsubject is standing next to the remote subject at a baseball stadium.

FIG. 3 is a block diagram of a system 300 in accordance with someembodiments of the present invention. The system 300 includes a firstlocal or studio camera 310 aimed at a local subject 312 from a firstangle. The first local video camera 310 might comprise, for example, aninstrumented hard camera that can be dynamically adjusted (e.g., via panand/or tilt motions). The first local video camera 310 might provideinformation about such dynamic adjustments directly to a first PersonalComputer (“PC”) 330 via a serial interface and/or linked fibertransceivers. The first PC 330 might be executing a renderingapplication, such as the Brainstorm eStudio® 3D real-time graphicssoftware package. Note that the rendering platform could instead beimplemented using an Apple® computing platform.

Similarly, the system 300 may include a first remote video camera 320aimed at a remote subject 322 (e.g., a guest standing in front of agreenscreen) from a first angle. The first remote video camera 320 mightcomprise, for example, a locokoff camera that transmits a remote videoHD-SDI feed directly to the first PC 330 over a fiber or satelliteconnection. Note that the first PC 330 might be co-located with thefirst local video camera 310 or the first remote video camera 320 or mayinstead be implemented at an entirely different location.

The first PC 330 may automatically adjust the received remote videoHD-SDI feed based on information about dynamic adjustments received fromthe first local video camera 310 (e.g., the image of the guest may beadjusted when the studio camera is tilted). As a result, the output ofthe first PC 330 may represent a tracked remote video foreground overgreenscreen video signal that may be provided to a first overlay engine340.

The first overlay engine 340 may also receive a local video HD SDI feed(including the studio background) directly from the first local videocamera 310. The first overlay engine 340 may then combine the tworeceived video feeds to generate an output video feed that creates animpression that the local subject and the remote subject occupy a sharedphysical space. Note that according to some embodiments, the first PC330 and the first overlay engine 340 may comprise a single device.

The system 300 may also include, according to some embodiments, a secondlocal camera 311 aimed at the local subject 312 from a second angle(different than the first angle). The second local video camera 311might comprise, for example, another instrumented hard camera that canbe dynamically adjusted (e.g., via pan and/or tilt motions). The secondlocal video camera 311 might provide information about such dynamicadjustments directly to a second PC 350 via a serial interface and/orlinked fiber transceivers. The second PC 350 might also be executing arendering application, such as the Brainstorm eStudio® 3D real-timegraphics software package. Note that according to some embodiments, thefirst and second PCs 330, 350 may comprise a single device.

Consider, for example, FIG. 4 which is an illustration 400 of a display430 including a local video feed received from a local or “studio” videocamera 410, the local video feed including a local subject 412. Thedisplay 430 also includes a remote video feed, from a remote videocamera 420, the remote video feed including a remote subject 422. Inthis example, an image 442 of the local subject 412 is displayed blendedwith an image 452 of the remote subject 422 to efficiently create theimpression that the local subject 412 and remote subject 422 occupy ashared physical space.

Referring again to FIG. 3, according to some embodiments the system 300may also include a second remote video camera 321 aimed at the remotesubject 322 (e.g., a guest standing in front of a greenscreen) from asecond angle. The second remote video camera 321 might comprise, forexample, another camera that transmits a remote video HD-SDI feeddirectly to the second PC 350 over a fiber or satellite connection. Thesecond PC 350 may automatically adjust the received remote video HD-SDIfeed based on information about dynamic adjustments received from thesecond local video camera 311 (e.g., the image of the guest may beadjusted when the studio camera is tilted). As a result, the output ofthe second PC 350 may represent a tracked remote video foreground overgreenscreen video signal that may be provided to a second overlay engine360.

The second overlay engine 360 may also receive a local video HD SDI feed(including the studio background) directly from the second local videocamera 311. The second overlay engine 360 may then combine the tworeceived video feeds to generate an output video feed that creates animpression that the local subject and the remote subject occupy a sharedphysical space. The two composite outputs from the first and secondoverlay engines 340, 360 might be routed to a patch panel 370 (andeither of the two angles might be selected for broadcast by anoperator). According to some embodiments, the system 300 furtherincludes a virtual operator station 380 that may facilitate interactionsbetween an operator and the two PCs 330, 350.

The system 300 may therefore provide an ability to have remoteguests/talent seamlessly immersed in a studio environment (or viceversa). For example, a remote guest might appear to be sitting in thestudio location alongside a studio host in the same camera shot.

According to some embodiments, the locked-off remote interview feed(over greenscreen) is fed from the first remote video camera 320 to theBrainstorm application executing at the first PC 330 as an HD/SDI liveinput, which may be mapped to a tracked plane in a virtual environment.The tracked plane of video may then be keyed over the encoded anddelayed studio camera shot from the first studio video camera 310 (e.g.,equipped with an encoded jib associated with a virtual setup) by aswitcher using a chroma keyer to complete the effect. Note that theoperator of the first remote video camera 320 may provide to therendering software information about the distance between his or hercamera to the subject and/or help calibrate the field of view (e.g., thewidth of the shot at the remote subject's distance).

FIG. 5 is a block diagram of a rendering engine 500, such as an engineexecuting on the first or second PCs 330, 350 of FIG. 3, in accordancewith some embodiments of the present invention. The rendering engine 500comprises a processor 510, such as one or more INTEL® Pentium®processors, coupled to communication devices 520 configured tocommunicate with remote devices (not shown in FIG. 5). The communicationdevices 520 may be used, for example, to receive a remote video feedalong with dynamic adjustment information about a local video camera andto transmit an adjusted remote video feed.

The processor 510 is also in communication with an input device 540. Theinput device 540 may comprise, for example, a keyboard, a mouse, orcomputer media reader. Such an input device 540 may be used, forexample, to enter information about a remote and/or studio cameraset-up. The processor 510 is also in communication with an output device550. The output device 550 may comprise, for example, a display screenor printer. Such an output device 550 may be used, for example, toprovide information about a remote and/or studio camera set-up to anoperator.

The processor 510 is also in communication with a storage device 530.The storage device 530 may comprise any appropriate information storagedevice, including combinations of magnetic storage devices (e.g., harddisk drives), optical storage devices, and/or semiconductor memorydevices such as Random Access Memory (RAM) devices and Read Only Memory(ROM) devices.

The storage device 530 stores a rendering engine application 535 forcontrolling the processor 510. The processor 510 performs instructionsof the application 535, and thereby operates in accordance anyembodiments of the present invention described herein. For example, theprocessor 510 may receive dynamic adjustment information from a localvideo camera associated with a local subject. The processor 510 may alsoreceive a remote video feed, from a remote video camera, the remotevideo feed including a remote subject. The processor 510 may thenautomatically adjust the remote video feed based on the dynamicadjustment information to create an impression that the local subjectand the remote subject occupy a shared physical space. The processor 510may then transmit the adjusted remote video feed to an overlay enginevia the communication devices 520.

As used herein, information may be “received” by or “transmitted” to,for example: (i) the rendering engine 500 from other devices; or (ii) asoftware application or module within rendering engine 500 from anothersoftware application, module, or any other source.

As shown in FIG. 5, the storage device 530 also stores camera data 600.One example of such a database 600 that may be used in connection withthe rendering engine 500 will now be described in detail with respect toFIG. 6. The illustration and accompanying descriptions of the databasepresented herein are exemplary, and any number of other databasearrangements could be employed besides those suggested by the figures.

FIG. 6 is a tabular representation of a portion of a camera data table600 in accordance with some embodiments of the present invention. Thetable 600 includes entries associated with different remote and localcamera pairs (designated as RC/LC in FIG. 6). The table 600 also definesfields for each of the entries. The fields might specify a cameraidentifier, a distance between a camera and a subject, tilt data, zoomdata, focus data, field of view data, etc. The information in thedatabase 600 may be periodically created and updated based oninformation received from, for example, camera operators and/orinstrumented cameras. Note that in the embodiment described with respectto FIG. 3, a table 600 associated with only a single remote/local camerapair might be needed by the rendering engine.

FIG. 7 is a block diagram of a system 700 in accordance with someembodiments of the present invention. The system 700 includes a studiocamera 710 aimed at a host 712 of a broadcast program. During aninterview, the studio video camera 710 might provide information aboutdynamic adjustments (e.g., movements of the studio video camera 710)directly to a rendering engine 730 (e.g., a virtual machine and keyer)via a serial interface and/or linked fiber transceivers. According tosome embodiments, the studio video camera 710 is a vertical camera mountwith integrated inclinometer. Prior to an interview, an operator of thestudio video camera 710 might use a tape measure to determine a distancebetween the host 712 and the studio camera 710 to help determine thewidth of the video frame (field of view) of the studio camera 710. Thisinformation might be entered into the rendering engine 730 to helpestablish an appropriate position of a remote guest 722 with respect tothe studio video camera 710.

The system 700 may also include a remote video camera 720 aimed at theremote guest 722 (e.g., standing in front of a greenscreen). The remotevideo camera 720 might comprise, for example, a locokoff camera thattransmits a remote video HD-SDI feed directly to the rendering engine730 over a fiber or satellite connection (e.g., a video/audio interviewuplink).

The rendering engine 730 may automatically adjust the received remotevideo HD-SDI feed based on information about dynamic adjustmentsreceived from the studio video camera 710 (e.g., the image of the guest722 may be adjusted when the studio camera pans from left to right). Asa result, the output of the rending engine 730 may represent a trackedremote video foreground over greenscreen video signal that may beprovided to an overlay engine 740.

The overlay engine 740 may also receive a studio video HD SDI feed(including the studio background) directly from the studio video camera710 (e.g., which includes a host 712). The overlay engine 740 may thencombine the two received video feeds to generate a combined output videofeed that creates an impression that the studio subject and the remotesubject occupy a shared physical space. With such an arrangement, studiosubjects may be immersed into a remote environment (or remote subjectsmay be placed into the studio environment) with significant flexibilitywhen producing interviews and analysis associated with a remote site.

Some embodiments described herein utilize capabilities of an encodedstudio camera system (with or without the addition of a greenscreen areaat the remote site or in studio). Note that an identical camera and lens(e.g., non-encoded or motion-controlled) could be used at the remotesite as compared to the studio. In some cases, camera data may besynchronized between sites via fiber or network connections. Thus,various combinations of equipment may provide different levels ofimmersion and/or interaction.

For example, in some cases no greenscreen might be used at either thestudio or the remote site. In this case, both cameras may shoot subjectsover the actual backgrounds and the two shots may be blended usingrendering virtual software along with in-studio camera tracking This mayallow a camera shot of the studio, where the studio subject can be oncamera, and when the camera pans to the left or right it will appear asif the remote set is actually present next to the studio set.

As another example, a green screen may by used at the remote site butnot at the local site. This may provide the added ability to place theremote subject “virtually” into the studio set. That is, it might appearas if the remote subject was actually standing in the local studio (andpossibly next to the actual local subject. As still another example, thelocal studio set may have a partial greenscreen area such that thestudio subject could walk from the actual physical set into the remoteenvironment on the same camera shot. Note that embodiments describedherein may require little or no added remote hardware and no additionalpersonnel, while substantially increasing the immersive options.

The following illustrates various additional embodiments of theinvention. These do not constitute a definition of all possibleembodiments, and those skilled in the art will understand that thepresent invention is applicable to many other embodiments. Further,although the following embodiments are briefly described for clarity,those skilled in the art will understand how to make any changes, ifnecessary, to the above-described apparatus and methods to accommodatethese and other embodiments and applications.

Although a single local subject and single remote subject have beendescribed in some of the examples presented herein, note that any numberof subjects may be blended in accordance with the present invention.Similarly, although a single local and remote site have been describedherein as examples, note that embodiments could blend together anynumber of locations. For example, an impression could be created that afirst football player located in New York and a second football playerlocated in Chicago are standing next to a studio host located inConnecticut.

Moreover, although specific hardware and data configurations have beendescribed herein, note that any number of other configurations may beprovided in accordance with embodiments of the present invention (e.g.,some of the information associated with the databases and enginesdescribed herein may be split, combined, and/or handled by externalsystems). Further note that embodiments may be associated with anynumber of different types of broadcast programs (e.g., sports, news, andweather programs).

The present invention has been described in terms of several embodimentssolely for the purpose of illustration. Persons skilled in the art willrecognize from this description that the invention is not limited to theembodiments described, but may be practiced with modifications andalterations limited only by the spirit and scope of the appended claims.

1. A method comprising: receiving a studio video feed, from a studiovideo camera, the studio video feed including a studio subject;receiving a remote video feed, from a remote video camera, the remotevideo feed including a remote subject; and overlaying the remote videofeed and the studio video feed to produce a composite video signal,wherein at least one of the studio video feed and the remote video feedare automatically adjusted, based on dynamic adjustments being made tothe other video feed, to create an impression that the studio subjectand the remote subject occupy a shared physical space.
 2. The method ofclaim 1, wherein at least one of the studio video camera and the remotevideo camera comprise an instrumented camera adapted to providesubstantially real-time information about dynamic adjustments made tothe instrumented camera.
 3. The method of claim 2, wherein the dynamicadjustments are associated with at least one of: (i) a panning motion,(ii) a tilting motion, (iii) a focal change, or (iv) a zoomingadjustment.
 4. The method of claim 2, wherein the remote video feed isreceived via a high definition serial digital interface signal.
 5. Themethod of claim 4, wherein the high definition serial digital interfacesignal is received via at least one of: (i) a fiber cable or (ii) asatellite transmission.
 6. The method of claim 2, wherein the remotevideo feed is automatically adjusted by a real time rendering platform,based on the dynamic adjustments made to an instrumented studio videocamera, to create the impression that the studio subject and the remotesubject occupy the shared physical space.
 7. The method of claim 6,wherein the impression created is that the remote subject is present inthe studio subject's physical space.
 8. The method of claim 7, whereinthe remote video feed includes the remote subject in front of asolid-colored background.
 9. The method of claim 6, wherein theimpression created is that the studio subject is present in the remotesubject's physical space.
 10. A system, comprising: a studioinstrumented video camera outputting (i) a studio video feed including astudio subject and (ii) data associated with dynamic adjustments; aremote video camera outputting a remote video feed including a remotesubject; a rending engine receiving the data from the studioinstrumented video camera and the remote video feed from the remotevideo camera and generating an adjusted remote video signal; and anoverlay engine receiving the adjusted remote video signal and the studiovideo feed and generating a combined output to create an impression thatthe studio subject and the remote subject occupy a shared physicalspace.
 11. The system of claim 10, wherein the system includes aplurality of studio video cameras and paired remote video cameras, eachpair being combined by a separate overlay engine.
 12. The system ofclaim 11, wherein the dynamic adjustments are associated with at leastone of: (i) a panning motion, (ii) a tilting motion, (iii) a focalchange, or (iv) a zooming adjustment.
 13. The system of claim 12,wherein the data associated with dynamic adjustments is provided fromthe studio video camera to the rendering engine via a serial signaltransmitted via fiber transceivers.
 14. The system of claim 13, whereinthe impression created is that the remote subject is present in thestudio subject's physical space.
 15. The system of claim 13, wherein theimpression created is that the studio subject is present in the remotesubject's physical space.
 16. A computer-readable medium storinginstructions adapted to be executed by a processor to perform a method,the method comprising: receiving dynamic adjustment information from astudio video camera associated with a studio subject; receiving a remotevideo feed, from a remote video camera, the remote video feed includinga remote subject; automatically adjusting the remote video feed based onthe dynamic adjustment information to create an impression that thestudio subject and the remote subject occupy a shared physical space;and transmitting the adjusted remote video feed to an overlay engine.17. The medium of claim 16, wherein the dynamic adjustment informationis received from an instrumented camera adapted to provide substantiallyreal-time information about dynamic adjustments made to the instrumentedcamera.
 18. The medium of claim 17, wherein the dynamic adjustments areassociated with at least one of: (i) a panning motion, (ii) a tiltingmotion, (iii) a focal change, or (iv) a zooming adjustment.
 19. Themedium of claim 18, wherein the remote video feed is automaticallyadjusted by a real time rendering platform, based on the informationabout the dynamic adjustments made to the instrumented camera, to createthe impression that the studio subject and the remote subject occupy theshared physical space.
 20. The medium of claim 13, wherein theimpression created is one of: (i) that the remote subject is present inthe studio subject's physical space, or (ii) that the studio subject ispresent in the remote subject's physical space.